When is a confirmatory randomized clinical trial needed?

Few clinical trials show important advantages for one therapy over another. When one does, a confirmatory trial will help decide whether the observation in the first trial was real. The need to replicate experimental results is fundamental in science. Systematic differences occur across clinical trials for a variety of reasons: differences in patient populations (geographically and over time), different patient referral patterns, differential application of entry criteria by investigators, and so on. Results vary across trials, just as they vary across patients within a trial, although intertrial variability is usually less than interpatient variability. If the differences among trials affect all treatment arms similarly, then treatment differences may be similar in different trials, but trial settings may well interact with treatment. The need for replication suggests that the answer to the question in the title is "Always." But carrying out a confirmatory trial has implications other than scientific ones. First, trials require commitments of time and resources. More important, the original trial suggested that the control therapy is inferior, and randomly assigning patients to receive a possibly inferior therapy is ethically questionable. Randomly assigning patients to therapy is the same scientifically as, say, randomly assigning plants to pots, but it is not the same ethically. Different people view trial results differently, in part for ethical, economic, and political reasons. So there are differences of opinion concerning the need for a confirmatory trial. In this issue of the Journal, Parmar et al. (7) suggest that differences of opinion can be explained from a Bayesian perspective (2) as differences in prior probability distributions of treatment effectiveness. They suggest using a skeptic's prior distribution to address whether a confirmatory trial is necessary. A confirmatory trial is recommended if, after updating this prior distribution on the basis of results from the original trial, the skeptic's resulting posterior probability of a clinically important treatment benefit is small. This Bayesian calculation is easy to make, and it can be very useful. Quantifying opinions (of real people as well as of "enthusiasts" and "skeptics") and showing how they are updated on the basis of empiric evidence can be pivotal in a decision process. The point of using posterior probabilities is not to dictate the final decision but to elucidate the decision process. Decision makers are not constrained to choose the option indicated by such an analysis, but if they choose otherwise, they should be able to identify which assumptions in the analysis are wrong, and they can redo the analysis with the repaired assumptions. Anyone considering a confirmatory trial will find the analyses of Parmar et al. instructive. Furthermore, enthusiastic and skeptical posterior probability distributions of treatment benefit would be useful additions to the discussion sections of clinical trial reports. In addressing the decision to confirm a trial, Parmar et al. restrict their consideration to data from the original trial. This is seldom the only available information. Consider the authors' example of the Cancer and Leukemia Group B (CALGB) trial of nonresectable stage III non-small-cell lung cancer (3,4). Prior to or during the initiation of the confirmatory trial (5), several other randomized trials (6-9) had compared radiotherapy plus chemotherapy with radiotherapy alone, although none used exactly the same chemotherapy regimen as did the CALGB. This other evidence could be included in a statistical analysis, and the Bayesian approach is ideal for synthesis. However, it would be wrong to regard patients from different studies as exchangeable and simply pool them. Hierarchical models (10-12) provide an appropriate alternative. In a hierarchical analysis, patients and trials are viewed as two different levels of experimental units. Patients are regarded as sampled from a trial's population, and trials are regarded as sampled from some larger population—a random effects model. The original trial is a sample of size 1 from this population, and this is so irrespective of the trial's size.